Stylized use of Photogrammetry in Videogames - Research
'Non-Conventional Use of Photogrammetry' - was the title given to my academic research paper I needed to conduct as my Graduation Work at DAE Howest, achieving considerable general praise and high mark (80%)
I always found Photogrammetry very powerful and under appreciated in the industry. I set out to answer 2 crucial questions in today's technological setting:
1. Is it feasible for a small or indie studio to utilize the technology without expensive equipment - if the do not require 1:1 photo realistic precision? - Can we incorporate the workflow into a modern PBR pipeline with ease?
2. Is it possible to stylize Photoscanned results to achieve a consistent looking artistic style that substantially deviates from its original version- thus creating stylized assets / environments with phtotgrammetry?
After a huge amount of work and learning I manged to answer both of these questions with a yes. The entire project was shot with a small tripod and an iPhone 11 Camera. I will let the results speak for themselves.
I was inspired by people like Trendt Boe Photogrammetry Reasearcher and Unreal specialist, who encouraged using the technology to acquire silhouette and shape, and use it with stylized textures, in his 2017 Brisbane Lecture to the UE4 Community.
An other key inspiration was the work of Grzegorz Baran, Material Expert at Ubisoft UK, who has an entire series dedicated to Photogrammetry and extensive knowledge on extracting data out of images.
The two main goals of my research were to find out whether it is feasible to utilize Photogrammetry for smaller studios or for Stylized projects. The entire project was shot with a tripod and an iPhone 11 Camera
Substance Designer Master Shader:
We optimize the mesh, project the high poly onto the low. Then we feed these maps into the Master Shader which can generate normal, roughness, height and displacement maps
To achieve stylize use of Photogrammetry we can ignore the obejct's scale, proportion or purpose. A toy car can be a real car, a twig a tree and plywood a roof beam
A simple breakdown of my Photogrammetry asset pipeline
I constructed several different iterations using the same assets I made with my pipeline. This shows the flexibility of the models and the way the player can view and interact with them
single shot render for a clicker adventure / mystery game, or for a fighting game background environment that uses a parallax camera movement. These scenes could benefit from very high quality baked lighting and dynamic background effects
Top down strategy game with minimal perspective distortion. This method would allow the use of lower resolution texture maps, more heavily decimated meshes and simple baked lighting to create vast open world maps with close to photo realistic results
In a setting like in this post apocalyptic wasteland - surface irregularities resulting from Photoscanning can become part of the surface detail, then the car mesh could easily be used as a vehicle in a racing or driving game
Using high resolution texture maps we can also effectively use the method to create scenes for FPS, third person and shooting games. These scenes can also benefit from high quality baked lighting and dynamic shadows, creating a unique environment
Open World setting: If we increase the texture resolution and refine each asset we create, we can populate a vast open world in many art styles, using photo realistic assets.
I constructed 4 vastly different scenes using the same assets, tweaking the Master Shader via the Substance to Unreal plugin. I wanted to push the shader as far as i can into the realms of stylization and see how different scenes can be made
Morning’: a fresh foggy morning scene set up in the mountains. The sun in waking up above long abandoned car. This scene benefits from crisp and noisy surface details, ambient fog and a green posterized variant of the T-shirt was used for the mountains
‘Nuked’ : A hellish scorched up wasteland resembling the aftermath of a recent nuclear detonation. All of the different effects are at use here
‘Fringe’: a mystical - possibly supernatural - desert location set in the remains of an old settlement the middle of nowhere. A large amount of different effects are used in this scene to create a unique look way different than the original scans
‘Lonely cottage’: A forgotten cottage in the mountains. A lonely but cozy feeling, in a quiet and ambient winter setting. The mosaic pattern effect is heavily used creating the impression that a lot of it was painted from patches
The Master Shader extracts info out of the original texture maps, thus models with a lot of unique detail will create equally unique stylized results, such as pointillist paint styles from rust or tree bark knots
Models with consistent patterns such as walls and flat materials will generate equally consistent stylized results, resulting in interesting vastly different surfaces that still work as intended - in this example as a wall texture
Bonus Renders: Outside of the scope of this project I applied my Master Shader pipeline to some high quality photos taken from various skulls. The results were stunning